1. Field of Invention
The present invention relates generally to interface devices between human users and computers, and more particularly to methods and systems that provide force feedback effects to the user.
2. Description of the Prior Art
Computer systems are used extensively in different industries to implement application programs including sophisticated computer controlled simulations, office-work tasks and interactive games. These interactive games have soared in popularity in the mass market of home consumers who use personal computers. A computer system running an application program or an interactive game typically displays a visual environment to a user on a monitor or other visual display device. The user interacts with this computer-generated environment to play a game, experience a simulation, or other virtual reality environment, by using various typical input devices, such as a keyboard, a stylus and tablet, a trackball, a mouse, a joystick, a gamepad, a steering wheel, foot or hand pedals, or the like, that is connected to the computer system providing the displayed environment. The host computer updates the application program in response to the user""s manipulation of an input device, and provides a corresponding feedback to the user via the display device. This virtual reality environment has also been extended to include audio speakers to provide the user with sound as well as visual displays to further enrich the user""s interactive experience.
Another extension of the virtual reality environment has been directed towards providing tactile feedback to the user, more generally known as force feedback. The inclusion of force feedback in input devices has transformed these devices from input devices to interface or physical interaction devices, because besides providing input to the host computer, they also provide physical sensations to the user. These physical sensations are generated in the physical interaction device in response to it receiving force feedback effect commands from the host computer executing a force feedback-enabled application program. Typically motors or other physical actuation devices are coupled to the interface object which is connected to the host computer to generate these physical sensations. Such physical sensations include the rumbling of the device when the user crashes her car in a virtual environment; mechanical jolts when the user drives over a speed bump; drag forces simulating aerodynamic drag on flying surfaces and xe2x80x9ckick backxe2x80x9d of the joystick to simulate the firing of a gun and the like.
The initial introduction of force feedback-enabled devices had focused on providing high fidelity effects in rather expensive and high-end products. The more current trend has been to expand the use of force-feedback-enabled devices into the higher volume consumer market. Typical prior art force feedback-enabled device manufacturer""s have focused their efforts on moving the control aspects of the force feedback effects to the device itself, by providing the devices with rather sophisticated local microprocessors and ample memory to allow the devices to store, manage, and compute all possible force effects locally. One reason for the large on-device memories has been the requirement of enabling the devices to be capable of playing back all the possible effect types that an application program has. This way, upon device initialization, an application program would download all its possible effect types to the device for on-device storage. A reason for including sophisticated on-device microcontrollers has been to avoid latency effects introduced by the time loop from the device to the host and back. Such time delays, although small, manifest themselves in the devices feeling choppy and rather delayed when playing back force feedback effects. Therefore these prior art approaches, upon the initialization of a device, download all the possible force feedback effect types to the device and store them in its memory for subsequent playback. Such prior art approaches are exemplified by devices manufactured based on technology from Immersion Corporation of San Jose, Calif. and described in various patents assigned to it as typified by U.S. Pat. No. 6,028,593, and devices manufactured based on technology from Microsoft Corporation of Redmond, Wash. and described in various patents assigned to it as typified by U.S. Pat. No. 6,036,495. The assignee herein, itself offers force feedback enabled devices based on the Immersion approach, such as the WingMan(copyright) Force joystick device. The limitations of these prior art approaches will be discussed below.
The DirectInput application program interface (xe2x80x9cAPIxe2x80x9d) in Direct X, from Microsoft Corporation has become a de facto standard API used to control these devices on the Windows-based platforms, which is by far the most popular operating system for home computers and which represents the largest potential mass market for such devices. DirectX is targeted towards game developers to allow them to directly and with least overhead possible, talk to the existing hardware, without having to explicitly consider hardware-specific parameters. It can be viewed as a minimal abstraction of hardware. Between the game (application program) and the hardware there are device drivers, which realize the abstraction for their specific hardware. DirectInput presents all connected gaming devices in a uniform matter to the game; the game can query the device through DirectInput for data, its capabilities and adjust its controls accordingly.
A hardware device vendor has to provide the following components to get their device running and accepted as a standard DirectX compliant force-feedback device:
a xe2x80x9cdriverxe2x80x9d, which is a piece of software that plugs into DirectInput, acts on requests from games to play/stop/change a force, and translates those requests into something the device understands under the best possible conservation of the fidelity;
a list of forces, registered in a system-wide database called xe2x80x9cregistryxe2x80x9d, describing what effects are possible, and what the effect""s limitations are in terms of timing resolution, additional complexity, advanced capabilities;
a force-feedback capable device which performs the commands sent to it by the driver
DirectInput""s role in this architecture is shown in FIG. 1. DirectInput""s role is simply to mediate and connect between the driver 114 and the device 116 and the application 110. The game""s requests pass through to the driver 114, after the initial setup, almost unmodified. The driver""s role in this architecture is merely a repackaging operation to strip down the application program""s request(s) down to what the device has. For example, gaming effects written with their time parameter in psec. may need to be converted to msec. DirectInput 112 does a minimal amount of error checking and parameter normalization and the game""s commands 110 end up at the driver 114 after a very thin layer of processing, which makes this approach very desirable in terms of performance, because by having only a small amount of processing, the application""s commands reach the device with the least possible amount of latency, and leave more processing time for other tasks on the host.
DirectInput""s force-feedback architecture was modeled after what is possible in devices, and devices were made with the DirectInput model in mind. Most devices have a more or less 1-to-1 correspondence in their internal workings to what DirectInput presents to a game. Most devices support a set of effects and a set of parameters for each effect. FIG. 2 shows this rather 1-to-1 correspondence between the game effects (those effects generated by the host application program) and the device effects (those effects downloaded to the device). The prior art devices provide sufficient local memory to allow it to download all possible game effects using their entire description to the device for storage and subsequent processing and playback. Typically games have anywhere from as few as two to three effects to as many as tens of effects. Upon initialization, the game (not shown) queries the device 210 for all possible effects 222, and downloads and stores these effects as on-device effects 212, thus resulting in a nearly 1-to-1 correspondence between the game and the on-device effects. Once the on-device memory is filled to its capacity, the device will typically reply with a xe2x80x9cdevice-fullxe2x80x99 command.
During the execution of the game, the driver 224 asks the device 210 for an effect, and passes it any additional parameters. The local device processor will then execute the game""s request and locally process the game effect(s) and play them back. As discussed above, this on-device approach has been pursued because the typical time loops involved in sending data from the device to the host for computation and its subsequent return to the device introduce undesirable latency effects which manifest themselves in the device behaving in an unstable or xe2x80x9cchoppyxe2x80x9d manner while playing back effects.
Most prior art force-feedback devices essentially use similar approaches. These approaches try to model the DirectInput architecture with a sophisticated combination of hardware and firmware. There are essentially two sub-categories for this implementation path. One is the Microsoft SideWinder path, and the other is the Immersion/I-Force chip-based approach. Both of them share the necessity for complicated hardware and smart firmware, which results in high costs to implement. The discussion below as summarized in Table 1 provides a brief overview of how these architectures perform.
The SideWinder series of force feedback devices pushes most of the processing onto the device. The device is responsible for handling everything. The management of effects; management of memory; calculation of forces and playback thereof are solely handled by firmware. The driver doesn""t have to do much besides adapt parameters to the capabilities of the device, i.e. stretch timings to values the firmware can handle, or correct out-of-range values for magnitudes or any of the other fields. After that, the driver asks the device for an effect handle, passing it the size needed; the device responds with either xe2x80x9cdevice fullxe2x80x9d, or returns a handle. Then, the driver sends out the serialized effect structure (i.e. converted from the DirectInput representation with memory blocks linked by pointers to a stream of bytes) to the device, and from there on directs the device to start/stop the effect with the given handle, or to dispose of it completely.
The Immersion approach is similar, however, with the difference that the device doesn""t have to do all the work. The driver on the host keeps track of the memory usage and allocates/deallocates blocks from the memory as required to fit in effect structures. It also checks all parameters for computability, serializes the effect structures and sends them to the device, telling it where to store the data. This approach is more robust towards transmission errors that can occur, because of the tight control of the host. If a transmission error happens and gets detected, the host can re-transmit the information it wants on the device. In the first approach, the host""s view of the device""s memory and the device""s view of its own memory can start to diverge in case of transmission errors.
The device, however, still has to do all the effect management, force computation and force output. However, the memory management is done in the host-based driver.
Typical prior art input devices interface to a host via either a serial or a game port, which require a vendor-specific driver to facilitate the interfacing. With the introduction of the Universal Serial Bus (xe2x80x9cUSBxe2x80x9d) standard to the computer hardware platforms, ideally the host would adapt to the device without the need for a vendor-specific driver.
USB defines a number of xe2x80x9cdevice classesxe2x80x9d, which define the behavior of devices belonging to this class. There was previously only one class dealing with input devices, namely the xe2x80x9cHIDxe2x80x9d (Human Input Device) device. HID devices are mice, keyboards and game controllers, for example. HID defines a rich set of descriptions which a device can use to describe its capabilities and limitations to the host. Ideally, the host would adapt to the device without the need for a vendor-specific driver.
HID does have one shortcoming, though, when it comes to force-feedback. As its name implies, it was designed for xe2x80x9cInput Devices,xe2x80x9d and the notion of force-feedback was not included. Recently, there has been work done to define an extension of the HID class, in the form of xe2x80x9cPIDxe2x80x9d (Physical Interaction Devices). Devices belonging to the PID class are devices that have a descriptor showing not only all the input usages (like with pure HID devices), but also output usages. xe2x80x9cUsagesxe2x80x9d is a term from the HID specification and denotes a bit of information describing some partial aspect of a device. PID usages extend the HID usages with output functionality such that a device can let the host know that it can play constant forces, sine force and a spring, for example. Following the HID concept, a PID device should ideally work by just plugging it in, where the device tells the operation system what it can do, and the operating system adapts to the device""s capabilities and limits, without the need for a specific driver.
Typical prior art PID devices need to have powerful processing capabilities. Just the storing of the PID descriptor needed for a typical device can take up to several hundred bytes, if not over a kilobyte of valuable memory space in the device. Moreover, since the PID usages define forces at a rather high level, in order to get from that high level to a level understandable by the motor(s) or actuator(s), a lot of processing (relatively speaking) is involvedxe2x80x94compared to what usual cheaper microcontrollers are able to do. Therefore, in order to make a PID device interface properly, a vendor-specific driver is almost necessary, since the PID usages do not allow for the definition of every limitation and special case a device could have, such as limited steepnesses for envelopes or other similar limitations. If a device should handle all of the possible effects cases properly, it needs even more processing power.
Therefore, as described above, the prior art PID devices rely on powerful local microprocessors, large required memories, and hardware-specific device drivers to be able to properly interface and playback force feedback effects. This constraint on the device to have a powerful local microprocessor and a large memory is driven by the need to (i) have a device that is capable of playing back all game effects, and (ii) avoid undesirable latency effects that may be introduced by the delay caused by the communication loop from the device to the host and back. The powerful local microprocessor as well as the large required memory adds complexity and increases the cost of such force feedback devices. The key to commercial success in the mass market for PID devices is in reducing the cost of the components and especially the electronic components of such devices. Less expensive parts yield less expensive devices. Therefore, there is a definite need for less expensive force feedback devices, that provide a realistic level of force feedback effects to their users.
The present invention provides a low-cost method and system for processing force feedback effects generated at a host for playback at a physical interaction device (xe2x80x9cPIDxe2x80x9d) by moving much of the processing of the force feedback effects from the device back to the host, and performing this in a manner that does not adversely affect latency effects or the load on the host processor. This is achieved by a novel combination of software and hardware techniques. In particular, the devices"" hardware design incorporates a low-cost microcontroller with limited processing capabilities, minimal memory and a low speed USB interface. The small memory also alleviates the need for buffer assignments and memory management. On the other hand, the device driver software residing on the host computer relies on the host""s powerful processor to perform the majority of the effect management and computation, as well as translating all the complex application program effect commands to a unique, small and simplified set of on-device commands for processing and playback on the device. This is made possible by removing the constraint that the device must at all times possess the full range of effects that an application program has, and the constraint that the device must store entire effects blocks.
In one aspect of an embodiment of the present invention, the device driver software program stores all the application program effects in its host-based Virtual Device, and thus portrays an infinitely capable device to the application program.
In another aspect of an embodiment of the present invention, the device driver software program classifies the application program effects into a simulated effects and native effects categories, where simulated effects are effects which are independent of the response of the device (such as a constant force), and native effects are force effects which are dependent on the input and response of the device (such as a spring force). Native effects are slated for local processing by the device, whereas simulated effects are slated to be driven by the driver using the host. Since the effects requiring input from the device are xe2x80x9cnative effects,xe2x80x9d their local processing will not introduce any latency effects. Moreover, since simulated effects are not dependent on the device""s position, any latency effects introduced by the delay in communication to and from the host, will not cause any instability issues and are not perceptible by the user.
In another aspect of an embodiment of the present invention, the device driver software program translates all the application program effects into a corresponding set of simplified effects. These simplified effects require substantially less space for storage, thus allowing for the reduction of on-device memory, which results in significant cost savings. The simplified effects include both simulated and native effects.
In another aspect of an embodiment of the present invention, the device driver software program periodically updates the device by transferring to it a subset of (n) simplified effects for on-device storage, which include at least one simulated effect, and which may also include as many as (nxe2x88x921) native effects. The transferred subset represents a reduced set of effects as opposed to the entire set of application program effects, and thus requires less on-device storage capacity and an hence provides additional cost savings. The number of effects transferred from the driver to the device are determined by the available on-device memory. The most important (i.e. perceptible by the user) effects are typically the ones stored on the device. When a new, stronger effect is requested, one less perceptible effect on the device can be written over.
When multiple simulated effects are provided at the same time, they are summed in the driver to further simplify and reduce the data which needs to be transmitted to the device. The at least one transferred simulated effect sent to the device is a vector sum of all application program simulated effects determined by the device driver software program. The remaining on-device memory is used to store a plurality of native effects, which represent the at most (nxe2x88x921) top prioritized native effects selected by the device driver software program. The periodic loading of the top (nxe2x88x921) prioritized native effects effectively implement a swapping operation of the highest (nxe2x88x921) top rated native forces.
These and other embodiments of the present invention are described in more detail in conjunction with the text below and attached figures.